Skip to content

CP-13581 linux-pkg: skip rebuilding windows-connector when its cached artifact is already up to date - #407

Open
vimleshmishra wants to merge 1 commit into
developfrom
dlpx/pr/vimleshmishra/11095495-a4c1-407f-8957-f85b9e3d7ab9
Open

CP-13581 linux-pkg: skip rebuilding windows-connector when its cached artifact is already up to date#407
vimleshmishra wants to merge 1 commit into
developfrom
dlpx/pr/vimleshmishra/11095495-a4c1-407f-8957-f85b9e3d7ab9

Conversation

@vimleshmishra

@vimleshmishra vimleshmishra commented Aug 5, 2026

Copy link
Copy Markdown
Contributor

Problem

The windows-connector package is rebuilt from scratch on every linux-pkg pipeline run, even when its source hasn't changed since the last successful build. Unlike other linux-pkg packages, windows-connector's build toolchain (nsis/mono/mingw-w64) lives outside our own source and can drift independently of any windows-connector code change — rebuilding on every unrelated trigger risks silently shipping a different toolchain combination without anyone deciding to re-certify it. This also masks the underlying dependency-drift bug (see dlpx-app-gate#4648) — the failure only surfaces when a buildserver's live apt mirror happens to have moved out of sync with the previously-pinned Artifactory dependency zips.

Solution

Add a new project.ext.connectorDebVersion in appliance/host/windows/build.gradle, bumped manually whenever connector source changes (independent of connectorVersion, which is bumped once per release regardless of whether the connector itself changed). Change config.sh's build() so the package is only rebuilt when that version doesn't have a matching artifact yet, reusing existing linux-pkg infrastructure rather than adding anything new:

  • Look up windows-connector's own current latest post-push artifact via get_package_dependency_s3_url — the same lookup every linux-pkg package already uses to fetch its own build dependencies.
  • If windows-connector_<connectorDebVersion>_all.deb already exists there, skip the build entirely and leave a .build_skipped marker file in the artifacts directory instead of trying to reuse/republish anything itself — devops-gate#4772 teaches the shared pipeline to recognize that marker, report a normal Jenkins NOT_BUILT result, and (for the multi-package orchestrator) resolve the existing artifact directly for downstream consumers.
  • Otherwise, build it (gradlew build + createDebPackage) as before. No explicit upload is needed — the shared post-push Publish stage already syncs the result to S3 and updates latest for us, exactly like every other package, when the build wasn't skipped.
  • The existence check fails soft: if the previous artifact (or its latest pointer) is ever missing for any reason, it just falls through to a normal build and republishes, rather than aborting.

This supersedes two earlier approaches tried in this PR: a bespoke release-de-images/windows-connector-cache/ S3 path, and (after that) writing a BUILD_STATUS file for a custom Slack notification line. Both were abandoned in favor of reusing existing latest-artifact infrastructure and Jenkins' own supported SUCCESS/NOT_BUILT result codes — no new S3 location, no new status plumbing.

Testing Done

Personal Jenkins controller testing: ops-jenkins's shared generate-jobs seed job embeds each job's pipeline script from a fixed branch at seed time rather than reading it fresh per build, so exercising in-flight devops-gate changes (like devops-gate#4772) required a personal developer Jenkins controller pinned to that PR's branch (JenkinsManager configure -j ops -g devops-gate=<branch>,linux-pkg). Two runs there cover both sides of this change:

  • Seed job that built the job definitions from the branch under test: generate-jobs #1
  • Build job that used those generated definitions to exercise this PR's cache-hit/skip path against a real windows-connector build: build-package/custom #2.build_skipped marker correctly detected, Finished: NOT_BUILT.

Notes to Reviewers

This PR depends on devops-gate#4772 to be safe to land — without it, a skipped build's .build_skipped marker is never checked, and the shared pipeline falls through to Publish with nothing to publish. Related, separately-tracked follow-ups:

  • dlpx-app-gate#4648: drop the pinned mono/mingw/nsis Artifactory zip, install those tools live from apt.
  • A git review pre-commit check (and/or CI check) in dlpx-app-gate requiring connectorDebVersion to be bumped whenever appliance/server/connector or appliance/host/windows source changes — without it, a merged windows-connector fix that forgets the version bump would be silently served from cache instead of shipping.

Tracked under epic CP-13579, alongside CP-13582 (app-gate).

@vimleshmishra
vimleshmishra force-pushed the dlpx/pr/vimleshmishra/11095495-a4c1-407f-8957-f85b9e3d7ab9 branch from f31f0e7 to d0a21e4 Compare August 5, 2026 14:03
@vimleshmishra
vimleshmishra marked this pull request as ready for review August 5, 2026 14:49
Comment thread packages/windows-connector/config.sh Outdated
Comment thread packages/windows-connector/config.sh Outdated
Comment thread packages/windows-connector/config.sh Outdated
@vimleshmishra
vimleshmishra force-pushed the dlpx/pr/vimleshmishra/11095495-a4c1-407f-8957-f85b9e3d7ab9 branch 3 times, most recently from be78e4c to f243840 Compare August 6, 2026 05:11
Comment thread packages/windows-connector/config.sh Outdated
Comment thread packages/windows-connector/config.sh Outdated
@vimleshmishra
vimleshmishra force-pushed the dlpx/pr/vimleshmishra/11095495-a4c1-407f-8957-f85b9e3d7ab9 branch from f243840 to 719b072 Compare August 7, 2026 05:01
@vimleshmishra
vimleshmishra force-pushed the dlpx/pr/vimleshmishra/11095495-a4c1-407f-8957-f85b9e3d7ab9 branch from 719b072 to 7ce82a9 Compare August 20, 2026 14:09
@vimleshmishra
vimleshmishra requested a review from sebroy August 20, 2026 14:27
Comment thread packages/windows-connector/config.sh Outdated

@prakashsurya prakashsurya left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I agree with seb's latest comment..

IMO, if we really want to keep the post-push build trigger on the app-gate git repository, then I think this is overly complicated.. and really, we want a way to denote the build as "skipped".. instead of the S3 file manipulation.. and do nothing when it's skipped, as the prior artifact will already be preserved "forever"..

With that said, since we want to tightly control when we do rebuilds of the connector, I might suggest an even more simple approach.. and disable the post-push trigger entirely, and instead rely on folks to manually trigger new builds.. then, we can carefully control when we rebuild it, and tie that into w/e "recertification" process we currently have (correct me if this doesn't make sense, I'm unfamiliar with the current certification process that's mentioned in the PR description).. and the build script can just be simple like it is today, and always do a build..

To me, this feels awfully complicated for what essentially equates to "do nothing, if connector version hasn't changed".. there's no need to copy the S3 artifacts around..

Comment thread packages/windows-connector/config.sh Outdated
Comment thread packages/windows-connector/config.sh Outdated
Comment thread packages/windows-connector/config.sh Outdated
Comment thread packages/windows-connector/config.sh
@vimleshmishra
vimleshmishra force-pushed the dlpx/pr/vimleshmishra/11095495-a4c1-407f-8957-f85b9e3d7ab9 branch 2 times, most recently from 514711f to 01deca0 Compare August 21, 2026 04:55
vimleshmishra added a commit that referenced this pull request Aug 21, 2026
@vimleshmishra
vimleshmishra force-pushed the dlpx/pr/vimleshmishra/11095495-a4c1-407f-8957-f85b9e3d7ab9 branch from 01deca0 to 2965e34 Compare August 21, 2026 05:13
@vimleshmishra vimleshmishra changed the title CP-13581 linux-pkg: build windows-connector on-demand, caching result in Artifactory third-party-local instead of rebuilding every run CP-13581 linux-pkg: skip rebuilding windows-connector when its cached artifact is already up to date Aug 21, 2026
vimleshmishra added a commit that referenced this pull request Aug 21, 2026
@vimleshmishra
vimleshmishra force-pushed the dlpx/pr/vimleshmishra/11095495-a4c1-407f-8957-f85b9e3d7ab9 branch from 2965e34 to baab544 Compare August 21, 2026 05:49
vimleshmishra added a commit that referenced this pull request Aug 21, 2026
@vimleshmishra
vimleshmishra force-pushed the dlpx/pr/vimleshmishra/11095495-a4c1-407f-8957-f85b9e3d7ab9 branch from baab544 to 4647325 Compare August 21, 2026 05:53
vimleshmishra added a commit that referenced this pull request Aug 21, 2026
@vimleshmishra
vimleshmishra force-pushed the dlpx/pr/vimleshmishra/11095495-a4c1-407f-8957-f85b9e3d7ab9 branch from 4647325 to b7229dd Compare August 21, 2026 06:40
vimleshmishra added a commit that referenced this pull request Aug 21, 2026
@vimleshmishra
vimleshmishra force-pushed the dlpx/pr/vimleshmishra/11095495-a4c1-407f-8957-f85b9e3d7ab9 branch from b7229dd to 46c827b Compare August 21, 2026 07:32
vimleshmishra added a commit that referenced this pull request Aug 21, 2026
@vimleshmishra
vimleshmishra force-pushed the dlpx/pr/vimleshmishra/11095495-a4c1-407f-8957-f85b9e3d7ab9 branch from 46c827b to c79ec9d Compare August 21, 2026 08:21
vimleshmishra added a commit that referenced this pull request Aug 21, 2026
@vimleshmishra
vimleshmishra force-pushed the dlpx/pr/vimleshmishra/11095495-a4c1-407f-8957-f85b9e3d7ab9 branch from c79ec9d to 194b461 Compare August 21, 2026 09:40
vimleshmishra added a commit that referenced this pull request Aug 21, 2026
@vimleshmishra
vimleshmishra force-pushed the dlpx/pr/vimleshmishra/11095495-a4c1-407f-8957-f85b9e3d7ab9 branch from 194b461 to c3f58bb Compare August 21, 2026 10:00
vimleshmishra added a commit that referenced this pull request Aug 21, 2026
@vimleshmishra
vimleshmishra force-pushed the dlpx/pr/vimleshmishra/11095495-a4c1-407f-8957-f85b9e3d7ab9 branch from c3f58bb to af2d7d0 Compare August 21, 2026 10:00
vimleshmishra added a commit that referenced this pull request Aug 21, 2026
@vimleshmishra
vimleshmishra force-pushed the dlpx/pr/vimleshmishra/11095495-a4c1-407f-8957-f85b9e3d7ab9 branch from af2d7d0 to 09739b9 Compare August 21, 2026 10:16
@vimleshmishra
vimleshmishra force-pushed the dlpx/pr/vimleshmishra/11095495-a4c1-407f-8957-f85b9e3d7ab9 branch from 09739b9 to c99d270 Compare August 21, 2026 12:13
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Development

Successfully merging this pull request may close these issues.

3 participants